This makes sense to me. I wonder if it's worth considering only storing the data we actually care about, and creating model fields for it, instead of the entire JSON blob? I guess we've hit times where we wanted extra data that we didn't store previously, but we could always re-query the API to get it, if needed. That would also cut down a good bit on the amount of data storage required.

That's a good question. One thing I'd mention is that having the json field as is helped me to understand the previous modelling, how it was used and the differences between the "duplicated" RemoteRepository objects.

I think it's a trade off between adding more work from our side (add these fields, populate them and maintain this code) vs. db storage.

There is a lot of API URLs in the response that most of them are useless for our case. The rest of the data looks useful, though. Maybe keep using a JSON field but removing these URLs before storing it?

I have a slightly preference to keep the whole json as is and rely on Postgres JSON internals, because seems simpler and safer.

Also, something to note is that the json field is independent from the Social Network, so it will definitely have different data.

Yea, I think at least trimming known useless data would be good. I mostly just want to get the data size reduced.

The other issue is, what is the API response format changes? It seems like that could wreck havoc on our side if we aren't parsing the incoming content to get the values we want.

Yea, I think at least trimming known useless data would be good. I mostly just want to get the data size reduced.

I replied too quickly last time. We are already parsing the relevant data and storing it as RemoteRepository fields outside the json field.

So, we may not require the json field at all. Instead, we may need to add some extra field in case we are missing something important.

The other issue is, what is the API response format changes? It seems like that could wreck havoc on our side if we aren't parsing the incoming content to get the values we want.

If the API response change, even if we are parsing the relevant fields, it may break. We do not have have a good way to protect ourselves here. To avoid this, we should pin to an API version on each service (e.g. https://developer.github.com/v3/media/#request-specific-version)

@ericholscher I updated the document with the latest changes in context. This is ready to merge IMO, and we can workaround some implementation details in the PR that implements this. I'm tempt to say that we can remove the json field completely if we are not going to use it anyways --we have all the data we need in DB model fields.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This is required. It's ready for review and merge, IMO. The implementation will take some time, tho.

I'm merging this since we are already implementing this and this document reflects almost exactly what we are doing. I just added a "Rollout plan" section with more info about this.